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ABSTRACT 

f The Vermont Vocational Special Needs Evaluation 

System is a statewide system for evaluating special projects for 
disadvantaged and handicapped students enrolled in secondary level 
vocational education programs. A program used for training 
individuals to serve as external evaluators for the System is 
presented. This document consists of four sections, which are: (1) an 
introductory section that provides a rationale for the training 
program, describes the selection procedures for external evaluators, 
and highlights the overall training program; (2) an overview of the 
Vermont Evaluation System, including the sequence of activities, 
local self-evaluation reporting and the guiding principles of the 
evaluation system; (3) the external evaluation activity, with 
discussion of the evaluator's schedule and the activities Recurring 
during the two-da^ onsite* evaluation , as well as a sample of an 
external evaluator report; and (4) an annotated bibliography of 
products developed during the operation of the Vermont ^evaluation » 
system project. (PN) 
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PREFACE 



For the past two years we, at the University of Vermont, have worked 
closely with the Vermont Division of Adult & Vocational Education in develop- 
ing a state system for evaluating disadvantaged and handicapped vocational 
education projects. The evaluation system was designed, field-tested and 
used in the evaluation of four projects during the first year of operation. 
Another outcome of the first year was the production of an evaluation 
system procedures manual for use by local education agencies.* 

The second y.ear of work focused on two activities; 1) the preparation 
of six persons to serve as external evaluators for the s.^ftem, and 2) the' 
evaluation of 11 vocational special needs projects. One result of these 
activities is the creation of the external evaluator training manual. 
"If this publication proves to be as helpful to the evaluation system as 
the local procedures manual, then it will be considered a successful product. 

A number of individuals made important contributions during the tv/o year 
effort. Robert Watson, State Consultant for Vocational Special Needs, was 
actively involved and supportive throughout this venture. His ongoing con- 
tributions and commitment to the evaluation system certainly reinforced our 
belief that the time and effort put into this activity was a worthy investment, 
James Trasier, Vocational Cooperative Education Coordinator at the Hartford 
Vocational Center, was a key figure in the development of the evaluation 
system. During the first year of operation, and while on sabbatical leave 

hhis document, which is provided to each district participating in a project 
evaluation, is available through the Vermont Division of Adult & Vocational 
Education in Montpelier. It can also be found in the ERIC Reproduction 
Service (ED 222-515). The reference for this publication is; Albright, L. 
& Frasier, J. The Vermont Vocational Special Needs Evaluation System : 
Procedures Manual for Local Education Agencies . Montpelier, Vermont: 
Division of Adult 4 Vocational Education, Vermont Department of Education, 
August, 1982. 



for graduate study* Jim played a major role in organizing the evaluation ^ 
system, served as the external ^valuator for four projects, and* co-authored 
the locaV procedures manual . Janet Ferguson, secretary in the vocational 
department at UVM, skillfully carried out the many and diverse activities 
involved in the day-to-day operations. We are especially grateful to 
Janet for her exceptional work in the typing of the project evaluation 
reports, the local procedures manual and the evaluator training manual. 

Special appreciation is extended to the six persons who served 
as external evaluators during the second year and who also provided helpful 
critique of the evaluation system; Lynn Baker, Timothy Donovan, Bob Grogan, 
Hallie Preskill, John McKeown and Laurie Waite. Recognition is due also to 
the many local steering committee members of the projects evaluated and to 
those members who participated in the follow-up telephone interviews about 
the evaluation experience. 

It is our hope that the vocational special needs evaluation system 

continues to enjoy the level of local, state and university support generated 

during the first two years of operation. With such endorsement the evaluation 

system should rightfully serve as an important contributor to the improvement 

of vocational programs and services for students with special needs. 

Leonard Albright 
Project Director 
June, 1983 
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INTRODUCTION TO EXTERNAL EVALUATOR TRAINING MANUAL 



Rationale for Evaluator Training Program 
Selection of External Evaluators 
Content of Training Program 
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/ INTRODUCTION 
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The information and materials presented herein are organized for use 
in preparing individuals to serve as external evaluators for the Vermont 
Vocational Special Needs Evaluation System. , The basic content for the 
evaluator training manual was developed over the latter months of 1982 and 
used in preparing six external evaluators during the first quarter of 1983. 
This document is, then, an outgrowth of the experience with an evaluator 
training program and is intended for use as a guide for conducting similar 
programs in the future. 

This introductory section provides the rationale for the training 
program; describes the considerations involved in selecting individuals 
for training as external evaluators; and outlines the content of the training 
program. Subsequent sections of this document offer a more detailed account 
of the training program "essentials". ^ 
Rationale for Evaluator Training Program ^ 

The training program for external evaluators was developed for three 
important reasons. First, since the Vermont special needs evaluation consists 
of a series of coordinated steps among local and state education agency • 
personnel and the external evaluator, it is imperative that the evaluator 
have a thorough understanding of the system before she/he arrives at the 
vocational center* to conduct the external evaluation. The fact that a 
local steering committee will have completed a self-evaluation prior to the 
external evaluator's visit provides ample rationale for the evaluator to be 
very familiar with how the evaluation system operates. 

Second, the various roles and responsibilities of the external evalu- 
ator prior to, during and following the two -day onsite evaluation also 
presents the need for advance preparation. Knowledge of the data that need 

r 



to be analyzed, facilitating local steering committee planning, and reporting 
evaluative information to the State are, for example, three evaluator activities 
which demand special consideration. 

^ The third reason for the training program is based on the Stated 
commitment to the continued operation of the evaluation system. In recognition 
of the critical role of the external evaluator, the State has invested in 
preparing a cadre of persons to serve as external evaluators. This initial 
investment will provide the State with a pool of trained and experienced 
evaluators to call upon in subsequent years. 
Select ion of E xte rnal Evaluators \ 

Since the vocational special needs evaluatiori system is a state- 
initiated and directed activity, final selection of external evaluators is 
nude by the State Consultant for Vocational Special Needs. Entering, into 
the selection decision are several important considerations. The individual 
selected should possess demonstrated leadership experience in education 
iind/or related human service areas (eg. rehabilitation), with prior experience 
in vocational and special education being preferred. In order to maintain 
neutrality, the individual should be from "outside" the present vocational 
education delivery system; but have an understanding of its operation. Other 
factors given prime consideration include; 1) positive human relations skills, 
2) ability to critical ly 'analyze data and contextual information, and 3) effective 
oral and written communication skills. 
Con ten t j) f 1 raj nj n£ Program 

* The*evaluator training program consists of two parts. The first part 
or session provides for an indepth review of .the Vermont Vecational Special 
Needs Evaluation System. The major phases and steps of the system are described, 
with frequent reference to the Procedures Manual for L ocal Education Agencies . 
It is recommended that' the participants read this manual prior to the training - 
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program so that pertinent questions and concerns can be addressed during the 
review session. 

The second part 'of the training session focuses on the external evalu- 
ation. The activities and responsibilities of the external evaluator are 
discussed indepth, along with materials to augment the evaluator's work. 
The autonomy of the external evaluated in conducting and reporting the 
evaluation becomes evident during this session. 

The agenda used in the first evaluation training program is shown 
in Figure I. This training program was a one-day session with the review 
section occurring in the morning and the external evaluation information 
presented in the afternoon. 
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. Figure 1 
AGf NDA 

EXTERNAL EVALUATOR TRAINING PROGRAM 



Session I:. "Orientation to Vermont Vocational Special Needs Evaluation System" 
AM A. Background and Purposes of the Evaluation System 

B. 'Indepth review of the LEA^Proc edu re Manga 1 f o r th ie Vermon t , 
Vocatidnarl Special Needs Eva1uation;^stjfn, including the' 
major phases of the system: 

• Orientation Session 
t LEA Self-Evaluation 

• External Evaluation 

t Local Project Improvement Plan r 

C. Guiding Principles of System 



Session II: "The External Evaluation" v . 

PM A, Rationale Behind Selection and Use of External Evaluator 

B. External Evaluator's Schedule 

C. Reflections on the Importance and Autonomy of External 
Fvaluator (Relationship to UVM, SEA, LEA) 

D. The Evaluator's Report: Format, Process & Timelines 
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OVERVIEW* OF EVALUATION SYSTEM 



Sequence of Activities 

Local Project Information 
Document 

✓ • 

Project Splf-Evalifation 
Report ^ 

Guiding Principles, of Vermont 
Evaluation System 
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r - ' OVERVIEW OF EVALUATION SYSTEM : 

* - • • «. 

This, section of the training 'manual presents information used in 
providing the external evaluators with a thorough orientation to the 
Vermont Vocational Special Needs Evaluation System. The major purposes, 
activities and principles of the evaluation system are presented, along 
with discussion of the self-evaluation report submitted to the State 
Division of Adult & Vocational Education by the local education agency. . 
Sequence of Activities 

An initial activity in the orientation session is a review of the major 
phases and steps involved in completing a project evaluation. However, before 
launching into this discussion, the dual purposes of the evaluation are noted. 
That is, the evaluation should assist local education agencies in improving 
vocational programs and services for special needs students, while also ful- 
filling the information needs of the .Vermont Division of Adult & Vocational 
Education. Particular attention is directed to the external evaluator's role 
in assisting the local education agency. 

The. major phases and steps involved in the Vermont system are presented 
sequencially through the use of the information provided in Figure 2 on the 
following page and by frequent reference to appropriate sections in the 
Procedures Manual for Local Education Agencies (eg. pages 2-4). 
Local Project Information Document 

' Following review of the major evaluation activities, the Session shifts 
to an examination of the special needs project information* obtained from 
the local steering committee. The types of information obtained about the 
local project are outlined in Figure 3 on page 9. The format and 
procedures used by the local committee in reporting this, information can 
be seen in Phase I - Section A of the Procedures Manual, pages 8-21. 



Figure 2 
SEQUENCE Of ACTIVITIES 
VERMONT VOCATIONAL SPECIAL NEEDS EVALUATION SYSTEM 



t ... ( 

I. Orientation • 

Step 1 - * Prerevaluation orientation meeting conducted by state H/D 
consultant 

, r 

II. Local Self-Evaluatton 

~ Step 2 - Preparation of project information document by the 
Local Evaluation Steering Committee 

Step'3 r Preparation of questionnaires by state H/D consultant 

\ 

Step 4 - Questionnaire administration^ tabulation/'arvalysijs .of 
results by^Local Steering Evaluation Committee, 

Hi. External Evaluation ^ " , 

Step 5 - External evaluator's visit:. > 
a. First day activities 
-* b. Second day meeting with Local 
, Evaluation Steering Committee . 

Step 6 - Preparation of External ^valuator's report 

IV. Local Improvement Plan m 

Step 7. - Preparation of Local Improvement Plan by .Local Evaluation 
Steering Committee 

Step 8 - Followrup visit by state H/D consultant to discuss and approve 
-Local Project Improvement Plan 

* 

\Step,9 - LEA response to local improvement reflected in project * 
- proposal submitted to state division of vocational 
education during next fiscal year 
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Figure 3 

LOCAL PROJECT INFORMATION DOCUMENT 

}. Narrative Description of the Project 

2. List of Project Goals and Enabling Objectives 

3. Student Identification and Referral: 

• criteria used to determine student eligibility 

• procedures used to identify students irv need of services 

4. Criteria Used to Select Students for Participation in the Project 

5. Individualized Planning Procedures: 

• procedures used in planning for individual needs of students 

• persons involved in planning for the individual needs of%students 

6. Monitoring Student/Project Achievement: 

• description of how student performance is monitored 

• description of how the Project's performance is monitored 

7. Student Termination/Follow-up: 

• procedures used to determine when a student no longer needs 
project services - 

% procedures used to follow-up on students no longer receiving project 
services but who are enrolled in regular vocational programs 

8. List of Project successes (at least 5) 

9. Description of school and community resources used by the Project 
and those available for use by the Project 

10. Numerical information on: 

• Number of special needs students" served last year 

• Number* of special needs students presently being served 

• Special needs studeht enrollment by vocational program area 
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Project Sel f-Eval uati on Report 

After seeing the types of project information obtained and the local 
reporting format, then the external evaluators receive a copy of the self 
evaluation report from their respective project evaluation site. At this 
point, time should be provided (eg. 20 minutes) for each evaluator to 
read through his/her local report. The intent here is only to acquaint 
the evaluator with the project to be evaluated. An independent and more 
indepth review of this material will need to be done by the evaluator 
following the training session and prior to the external evaluation. 
Guiding Principles of System 

The morning orientation session can be concluded by reviewing the 
guiding principles of the Vermont Vocational Special Needs Evaluation 
System. This focus should be helpful in reiterating the importance of 
the external evaluator role and the value attached to a responsive 
approach. A listing of the guiding principles is provided in Figure 4. 
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Figure 4 

G'UIDING PRINCIPLES OF THE VERMONT VOCATIONAL 
SPECIAL NEEDS EVALUATION SYSTEM 



1. That the system is state initiated and controlled, but 
it must be responsive to both SEA and LEA needs; that 
is, responsive in terms of usefulness of information 
obtained from the evaluation. 



That local use of evaluation results will require active 
LEA involvement in the process and with the external 
evaluator and state consultant. 



3. That an evaluation of a special needs project which 
focuses on providing services to students and teachers 
in mainstream vocational education must obtain evaluative 
information from these folks (ie. the primary audiences 
affected by the project). 



4. That an external evaluator should not only share her/his 

observations of the project with key LEA personnel, but, f*R# 
also assist them in formulating an appropriate plan of 
action for improvement. 



5, That both LEA and SEA must understand and endorse the 
LEA project improvement plarj. 
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SECTION III 

THE EXTERNAL EVALUATION 

External Evaluator's Schedule 
Activities During First Day 

r 

Activities During Second Day 
The External Evaluation Report 



. 19 

12 



ERJC 



THE EXTERNAL EVALUATION 

The second part of the training program is devoted primarily to the 

f 

external evaluation. The evaluator's schedule is reviewed, with a focus 
on the activities that occur during the on-site evaluation. The format 
of the evaluation report is also discussed and reporting timelines and 
procedures are established. 

External Evaluator's Schedule 

As indicated in Figure 5, the external evaluator's schedule at the 
project site involves -two intensive days o,f information gathering, analysis 
and reporting. A substantial portion.of this time is spent in inter- 
viewing various individuals at "the vocational center and in working with 
the local steering committee on the meaning and use of the evaluation 
findings. 

A ctivities During First Day 

The evaluator schedule during the first day essentially consists of 
three activities; 1) collection, verification and analysis of data from 
faculty, staff and student surveys in the morning, 2) interviews with 
students, faculty and administrators during the afternoon, and 3) indepth 
review and synthesis of information collected and preparation of reporting 
to local steering committee. This* latter activity needs to be done during 
the evening of the first day. 

However, before these first day activities begin, the evaluator is to 
meet with the local steering committee to finalize the two-day schedule. 
This includes arranging Interview times with various personnel, securing 
space for independent work to be done, and setting the meeting time and 
location for the session with the local steering committee members during 
the second day. Since the second day session with the steering committee 
1s when the findings are reviewed, recommendations are discussed, and- 

13 . 20 



Figure 5 

SAMPLE SCHEDULE FOR EXTERNAL EVALUATOR 



TIME FIRST DAY'S ACTIVITIES 

8:00 - 8:30 AM Evaluator meets with local steering committee members 

to finalize schedule for the evaluation and make all 
necessary arrangements. 

8:30 - 11:30 AM Evaluator verifies returned questionnaire percentages 

conducts preliminary analysis of responses, reviews 
related self-evaluation documents and notes areas in 
need of further probing. 

12:30 - 4:00 PM Evaluator meets with special needs staff to discuss 

project operation. 

Evaluator Interviews: 

• Two students (one receiving services; one 
not presently receiving services but has 
in the past) 

• Two vocational teachers (one who has student 
receiving project services; one who pre- 
viously had a student receiving project 
services) 

• Area guidance coordinator and vocational 
director 

Evening Evaluator reviews all data and information received 

and prepares for presentation of observations, find- 
ings and recommendations to local steering committee 
members . 



8:30 - 11:30 AM 



SECOND DAY'S ACTIVITIES 



Evaluator meets with all members of local steering 
committee to discuss findings, observations and 
recommendations and talk about the development of 



local improvement plan 



12:30 - 5:00 PM Evaluator time to prepare external evaluation 

report. 
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initial planning for program improvement occurs, the evaluator should, 
during the initial meeting with the steering committee on the first day, 
stress the need for all steering committee members to participate in the 
second day review and planning session. 

Collecting, Verifying and Analyzing Survey Data 

Before, or during the initial meeting with the steering committee, the 
evaluator is to collect the cumulative results from the surveys that were 
administered and tabulated by the local committee. Ttfis local steering 
committee activity is based on the instructions provided in Section C, 
pages 32-41, of the Pr ocedures Manual for Local Education Agencies . 

Upon receipt of the cumulative survey data and the individual survey instru- 
ments, the evaluator should verify the return rates of the various groups sur- 
veyed; examine the cumulative data; spot-check the responses on individual 
instruments for type and quality of information offered; and begin to formulate 
areas of questioning for the interviews in the afternoon. 
Conduntin<j Intervietfn 

Following review of the cumulative data and related project information, 
the evaluator begins the interviewing process. At a minimum, interviews are 
to be conducted with one special needs staff member, two vocational teachers, 
two students, the area guidance coordinator and the vocational director. 
Additional information on the types of individuals to be interviewed appears 
in Figure 5. The selection of these persons is a local responsibility, but 
the selection procedures and scheduling arrangements are to be discussed 
during the evaluator's initial meeting with the steering committee. 

The individual Interviews usually run no longer than 30 minutes and ^end 
to be semi-structured; that is, the evaluator may have specific areas of 
questioning, but the evaluator and the person being interviewed may spend 
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more time discussing some areas than others. The evaluator's intent is 
to see if the interview information is consistent with the survey results 
and to probe more deeply into areas of M fuzzines$ H or uncertainty. 
Reviewing and )i\imi 'ziny Findings 

The evening of the first day is the evaluator's time to review and 
synthesize the data and organize this information for presentation to the 
steering committee the following day. The format most often used in 
organizing the presentation consists of three major sections; 1) areas of 
project strength, 2) areas in need of strengthening, and 3) recommendations 
for project improvement. 

Since the observations of the evaluator are to be based on the data 
and information received from the surveys and interviews, and that hoth 
observations and "back-up" data need to be communicated to the steering 
committee, the evajuator should consider organizing a graphic presenta- 
tion of the data. A graphic display of the cumulative ratings from the 
survey instrument is one technique that many evaluators have used. An 
example of this display appears in Figure 6. This particular visual 
presentation of the data has been found to be very helpful in generating 
discussion among evaluator and steering committee members. 

Activ ities During Second Day 

The second day consists of two major activities. The first activity 
is the evaluator session with the steering committee. This important sessi< 
is when the evaluator shares her/his observations of the project operation, 
seeks committee member responses to these observations, and both parties 
explore strategies for improving the vocational special needs project- 
In addition to viewing a graphic display of the data and "teasing-out" 
observations from these data, the steering committee can record key points of the 
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discussion by using the review session form on pages 47 and 48 in* the 
Procedures Manual for Local Ed ucation A gencies. 

A number of areas for program improvement often emerge during this ^ 
session. To help the steering committee with its local planning, the evalu- 
ator may need to work with the committee on establishing priorities for • 
future action. * m 

The second activity of the day is to prepare the external evaluation 
report. This report is to be submitted to the State Cor|ultant for 
Vocational Special Needs within one week following the onsite evaluation. 
After reviewed and approved by the State Consultant, copies are sent to 
the local steering committee. The committee is then responsible for 
preparing a local project improvement plan. .Once completed, this plan 
is also presented to the State Consultant for review and acceptance. 
The External Evaluation Report 

The external evaluator's report consists of six major sections. 
These are: 



1 . Cover page 



2 . Purposes of . 
Evaluation 

3. Evaluation fli&thods 
and Procedures 



4 . Description of 
th e~Projec t 

5 • Swmwrx _9_L fj_n_di H2_s 



6. Recommendations 



Identifies the project evaluated by title, 
the person responsible for preparing the 
report (1e, the external evaluator) afhd 
the date that the report was submitted to 
the State Division. 

Presents the rationale for the evaluation 1 
and the dates of the external evaluation. 

Summarizes the self-study procedures used, 
followed with a description of the procedures 
used during the external evaluation phase. 

Provides an overall descriptive summary of 
the project. 

Lists projects strengths and areas in need 
1n need of strengthening, along with support- 
ing data for each observation noted. 

The evaluator provides specific recommendations 
for project Improvement; based on project obser- 
vations, Information reported 1n self-evaluation 
materials, and Information discussed during 
review session with local steering committee. 
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To illustrate this reporting format and style, a copy of an external 
evaluator's report from one project evaluated in Spring, 1982 is provi 
on the following pages. 

When submitting this report to the State Consultant, theevaluator 
should also include the local documents received during the external 
evaluation (eg. project description, cumulative ratings sheets). 
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SAMPLE EVALUATION" REPORT 



*■ 

SPECIAL NEEDS VOCATIONAL PROGRAM 
EVALUATION REPORT 



FOR 




(Project Title) 

at the 

ocational Center Title) 



SUBMITTED TO 



State Consultant for Handicapped and pisadvan taged 
Vocational Education Programs and Services 

Division of Adult and. Vocational Education 
* Vermont Department of Education 



PREPARED BY 

S 



External Evaluator 



(Evaluator's Signature) 
June 30 » 1932 
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PURPOSES OF THE EVALUATION 



The information in this report is based on an evaluation of the 

Project. This evaluation 
was conducted at the Center on June and , 1982, by an independent, 
external evaluator for the purposes of: 

1. Assisting the Center's faculty and the Project's coordinator 
in their efforts to continually improve. the delivery of services 

to mainstreamed vocational special needs students. y 

2. Furnishing the Center with evaluation information for use in 
developing a Local "Program Improvement Plan for presentation 

to the State Consultant for Handicapped and Disadvantaged Programs 
and Services as provided for in Phase Til of the Division's 
Evaluation System for Vocational Special Needs Programs and Services 
Manual. 

3. Providing information needed by the State Division, of Vocational 
Education to determine the effectiveness of the Project, 

as mandated by the 1976 Vocational Education Act Amendments (PL 94-482) 



EVALUATION METHODS AND PROCEDURES 



The evaluation of the Project was implemented at the 

Area Vocational Center by using the self-study materials prescribed in the 
Division's Evaluation System for Special Needs Programs and Services. After 
completion of the self-study, the visit by an external evaluator was used to: 

1. Review the Project's self-study documents and materials as shown 
in Attachment A, B» C. 

2. Verify that State Education Agency required data collection. and 
analysis procedures were satisfactorily completed. 

3. Interview the vocational center director, two School faculty 
and four Center faculty, two Project students and the 
Project's coordinator. 

4. Conduct an in-depth meeting with the Center's Local Evaluation 
Steering Committee to review the evaluator's findings; identify 
Project strengths and areaa in need of strengthening; and, to 
formulate recommendations concerning the Project. 
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DESCRIPTION OF THE PROJECT 

•'; ' | 

The Project is a two part effort designed to serVe the 

vocational needs of hearing impaired students enrolled at the 
Area Vocational Center and the School . 

The first part of this Project provides for the integration of 
hearing impaired students from the School into regular vocational 

education programs at the Center. This part of the Project 

requires the Project coordinator to devote about two (2) hours of each school 
day to: 

a) transporting students to and from the ; 

b) securing classroom interpreter services for students who 
are enrolled at the , Center; 

c) acting as a liaison person between students, the 
School administration, the faculty and the 
Centers administration in order to provide for the smooth integration 
of students; and, 

d) ' occasionally discussing each student's overall progress with his/her 

vocational instructor. 

During the 1981-82 school year, six (6) students participated 

in regular vocational education programs at the Center. 

The second part of this Project requires the Project coordinator to 
devote about 2/3 of eaph school day at the School to teach: 

a) three pre-vocat ional and/or vocational courses, one time each day 
for one hour each, in photography, graphic arts and industrial arts. 

b) one general math course, one time each day for one hour; and, 

c) one general english course, one time each day for one-half an hour* 
In addition, the Project provides for one orientation tour of selected 

students to the Center each school year; encouraged 

students to enroll at the Center through the activities of the Project's 
coordinator; and, conducts follow-up surveys of students who have 

attended the , 

^ It should be noted that this evaluation focused on the Center's 
part of jthis Project. 



SUMMARY OF FINDINGS 

PROJECT STRENGTHS 

1. The faculty and administration of the School and the 

Center have a good understanding of the Project and its goals. 

i The evidence for this finding can be found in reviewing the 
mean scores of individuals surveyed in Attachment B, Questions 
01 and #13 respectively. 

\ 
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PROJECT STRENGTHS (Continued) 

2. The faculty and administration of the Center and 
the School believe that this Project is needed to help 
students succeed in their pre-vocational and/or vocational program. 

• Individuals surveyed believe this Project is "needed" to 
"definitely" needed for helping students to succeed 

"in their vocational program. 

• Individuals surveyed believe that more than 50 percent of 

students enrolled in this Project would not be able 
to succeed without the special services provided by the 
Project (Attachment B, Question #17). 

3. The Center's faculty and administration believe this 
Project has been "successful" in helping students succeed in 
their regular vocational education programs at the Center > 
(Attachment B. Question #3). 

• The faculty and administration believe that the Project's 
goals have been "implemented" to "fully Implemented" 
(Attachment B t Question 014). 

• The faculty and administation believe the Project's goals 
have been "well" implemented (Attachment B, Question 015) . 

4. Virtually all of the Center's faculty believe that this 
Project is making a critical difference to an student being able to 
pass his/her vocational program at the Center (Attachment B f 
Question #16 and Attachment C t #16). 

5 # students and faculty interviewed by this evaluator 

believed the most valuable aspect of this /Project was that students 
were able to be in an ^educational setting with "hearing" people. 

/ 

• Additional support for this finding can be found in Attachment A t 
Section I t Page 11. 



PROJECT AREAS IN NEED OF STRENGTHENING 

The predominate finding of this evaluator was that the Project's planning 
and operational processes need to be strengthened in, order to more adequately 
serve the needs of hearing impaired students enrolled at the Center 
and in pre-vocational and/or vocational programs at the School 
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PROJECT AREAS IN NEED OF STRENGTHENING (Continued) 

l 9 xhe School's faculty appears to be uncertain about the 

Project's ability tq help students succeed in their pre-vocational 

and/or vocational program. 

§ The faculty could not state that this Project has 

been successful in helping students to succeed in 

their vocational program (Attachment B, Question 12). 

• Of the faculty surveyed, only two (2) indicated that 
this Project is making a critical difference to an 

student being able to pass his/her vocational program, three (3) 
answered "No" that it is not making a critical difference, and 
four (4) could not say if it was making a critical difference, 

f. Some faculty offered written comments that questioned 

the need tor this Project based on past Project performance of 
students agjfe others questioned how much students are being 
provided with stated Project services (Attachment C, Question #3 
and Question #18) • 

2. The Project appears to be viewed by the and 

faculty members as primarily being an interpreter service, 
which is staffed with a Project coordinator who is available to help 
with problems concerning students at the * Center, 

• Interviews conducted by this evaluator with students, 

and School faculty members, and later confirmed by the 

Local Evaluation Steering Committee members, revealed a general lack 

of knowledge among the two faculties that this Project involves 

more than an interpreter/coordinator service at the Center. 

• Discussion with the Local Evaluation Steering Committee indicated* 
that the School Part of this Project has come to be * 
regarded as a "regular program 11 and not as a special project funded 

* by the State. 

3. Xhe exceptionally high turnover rate of interpreters was the most 
commonly expressed concern of students, faculty and administration. 

• This was apparent in interviews conducted by this, evaluator with 
students, faculty members of both schools, administration and the 
discussions during the Local Evaluation Steering Committee Meeting , 
conducted by this evaluator. 

• Additional evidence for this finding can be found in Attachment C, 
Question 118. 
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PROJECT AREAS IN NEED OF STRENGTHENING (Continued) 



4. The procedures used to identify, select and refer students for 
participation in the Project appeared to not be working well. 

• The meaned score of faculty surveyed indicated that students 
in the Project may not be meeting stated selection criteria 
for enrollment in the Project (Attachment B 9 Question #6). 

• Only 3 of 6 students completed a full year of study in their 
vocational program at the Center this year 
(Attachment A, Page 4) . 

5. A formalized planning process that Involves and 
faculty of students enrolled in this Project was found to be lacking. 

• The faculty's involvement in planning for the services provided 
by the Project ranged from "no involvement" to "very little" 
invplvement (Attachment B. Question 19). 

6. Communications between the and faculty with 

the Project coordinator concerning individual student progress was found 
to be almost non-existant . 

• The faculty of the School is seldom Involved and the 

faculty only "occasionally" involved in such 
discussions (Attachment B, Question #12). 



RECOMMENDATIONS 



1. Key personnel from both schools (e.g. the Center's 
director, the School's principal, the Project coordinator and 

faculty members from both schools) should meet for the purposes of discussing 
and establishing procedures to strengthen: 

A) student identification, referral and selection; 

B) program planning to meet the unique needs of individual students 
that involves vocational faculty at the School and at the 

Center; and, 

C) cowunication between the vocational faculty, the faculty 
and the Project coordinator concerning the monitoring of students 
in this special project. 
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RECOMMENDATIONS (Continued) 



2. Specific activities be planned and implemented to decrease the 
turnover rate of interpreters for the students enrolled in this 
Project at the Center. 

3. Specific activities be planned and implemented in order to bring 
about a more complete understanding of the Project's activities 
within the and faculty members affected by this Project. 

4. The Part of this Project was not a focus of the present 
evaluation. Therefore, if is recommended that a follow-up evaluation be 
conducted so that a more compile assessment of the Project can be obtained. 
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SECTION IV 



ANNOTATED BIBLIOGRAPHY OF PUBLICATIONS 
FROM UVM-SDE EVALUATION PROJECT 
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Annotated Bibliography of Products Developed During 
The Vermont Vocational Special Needs Evaluation 
System Project (1981-1983) 



Albright, L. DeyeJ ojxnenX of a Stat e System for Evaluating Disadvantaged 
and Handicapped Vocational Education Projects . Final Project Report. 
Montpelier, Vermont: Division of Adult & Vocational Education, Vermont 
Department of Education, August, 1982. 27 pages. (ERIC Document 
Reproduction Service No. ED 222-510) 

This final' report of a first year project describes the major procedures 
employed in developing, field testing and using a system for evaluating 
vocational special needs projects in Vermont. Also included is a listing 
of project outcomes and products and recommendations for further improve- 
ment of the state evaluation system. 

Albright, L. The Vermont Vo cational Special Needs Eval uation Sy stem: Training 
Manual for External Evaluato^ s. fiontpelier, Vermont: Division of Adult 
& Vocational Education, Vermont Department of Education, May, 1983 - 30 pages. 

This document consists of procedures and materials used in the training 
of external evaluators for the Vermont Vocational Special Needs Evaluation 
System during the 1982-83 school year. 

Albright, L. & Frasier, J. The Vermont Vocational Special Needs Evaluation 
System: Procedures Manual for Local Education Agencies . Montpelier, 
Vermont: Division of Adult & Vocational Education, Vermont Department 
of Education, August 1982 - 68 pages. (ERIC Document Reproduction 
Service No. ED 222-515) 

A procedures manual for local education agency (LEA) use in organizing 
and conducting the state-sponsored vocational special needs program 
evaluation system. The three major phases of the evaluation are 
described. The first phase is a self-evaluation conducted by local 
education agency personnel and reported to the State. The second 
phase is an external evaluation, conducted by an evaluation specialist 
who spends 2 days at the respective LEA. The third phase is the develop- 
ment of a project improvement plan, based on the outcomes of the evalu- 
ation activities. 

Ericson, A.; Watson, B. ; Frasier, J. & Albright, L. Development o f a State 
Sys tem_ f or Evalu ating Dis advantaged and Handicapped V ocational Educat ion 
Projects. Pa pe represented at the American Vocational Association Con- 
vention," St. Louis, Missouri, December, 1982. 16 pages. 

This paper describes the procedures used in the development and review of 
the Vermont Vocational Special Needs Evaluation System. Also presented 
are; 1) the follow-up instrument used to interview LEA personnel who 
participated in project evaluations during the 1981-82 school year; and 
2) a chart which highlights the information obtained from the Vermont 
evaluation system. 
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Frasier, J.R. An Ana lysis of State Systems for Evaluating Vocational Education 
Programs and Services for Handicapped and Disadvantaged Students . Advanced 
Graduate Thesis. Burlington, Vermont: College of Education and Social 
Services, University of Vermont, April 1983. 151 pages. 

This thesis is a reporting of a study which examined the status of voca- 
tional special needs program evaluation in 13 states. The analyses con- 
sisted of a* review of each state's evaluation materials and follow-up 
telephone interviews with 13 state consultants for vocational special 
needs. The findings provide a multi-state reading on the current level 
of activity in the vocational special needs program evaluation arena 
and suggestions for future research and development are offered. The 
appendices to this report contain the materials and instruments used 
in the inquiry. 

Frasier, J. & Albright, L. Toward Building a Special Needs Evaluation Component 
Into the Quality Assessment System (QAS) . Report submitted to the Vermont 
Division of Adult & Vocational Education, October 30 t 1981. 6 pages. 

This report provides a series of suggestions for integrating a special 
needs focus into the existing state vocational education evaluation 
system; that is, the Quality Assessment System (QAS). Specific areas 
of the QAS in which the special needs component could be infused are 
noted and discussed. 
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